According to statistics, a large part of large amount of malicious codes belong to deceptive malicious codes. They usually use icons which are similar to those icons commonly used softwares to disguise themselves and deceive users to click to achieve the purpose of communication and attack. Aiming at solving the problems of low efficiency and high cost of traditional malicious code detection methods based on code and behavior characteristics on the deceptive malicious codes, a new malicious code detection method was proposed. Firstly, Portable Executable (PE) file icon resource information was extracted and icon similarity analysis was performed by image hash algorithm. Then, the PE file import table information was extracted and a fuzzy hash algorithm was used for behavior similarity analysis. Finally, clustering and local sensitive hash algorithms were adopted to realize icon matching, designing and implementing a lightweight and rapid malicious code detection tool. The experimental results show that the designed tool has a good detection effect on malicious code.

Since it's hard to analyze the cryptographic procedure using method of property scan or debugging for the variety and different implementation of cryptographic algorithms, a method was proposed based on library function prototype analysis and their calling-graph building. Library functions prototype analysis is analyzing cryptographic algorithm knowledge and library frame knowledge to form a knowledge base. Calling-graph building is building a calling-graph that reflects the function calling order according to parameter value of the functions. Finally cryptographic algorithm knowledge and library frame knowledge on the calling-graph were extracted. The method discriminated common cryptographic algorithm almost 100%, and it could not only recover cryptographic data, key and cryptographic mode, but also help to analyze the relationship between more than two cryptographic algorithms dealing with the same data. The method could be used to analyze Trojan, worm and test whether the library is used correctly.

For malware analysis and cipher application security validating, identification and filtration of cryptographic function from binary code has great significance. The memory operation and basic block loop characters were analyzed from cryptographic functions. According to the theory of binary data's information entropy, the characteristic of high-entropy of cryptographic algorithms was verified, a cryptographic functions filtration model was constructed based on dynamic loop entropy, and the hybrid (dynamic and static) method was adopted to reconstruct dynamic memory data in basic block loop. The experimental result shows that the filtration model has reliability and veracity.

According to the software network communication behavior, a reverse analytical method based on session association was proposed in this paper. The method restored the network traffic communication session and Application Programming Interface (API) sequence session produced by software firstly, then associated the sessions restored. Through the association, a direct mapping was built between two kinds of software network behavior analytical methods based on execution trace analysis and network traffic analysis respectively. The prototype system was designed and completed. Based on the system, the function call list was extracted. The reverse analytical method based on session association makes the reverse analysis of software network behaviors fast and convenient.

Reverse parsing unknown network protocol is of great significance in many network security applications. Most of the existing protocol reverse parsing methods can not handle the encryption protocol or get the semantic information of the protocol field. To solve this problem, a network protocol parsing technique based on dataflow analysis was proposed. According to the data flow recording tool developed on Pin platform, it could parse the network protocol with the aid of the dependence analysis based data flow tracking technology, as well as obtain the protocol format and semantic information of each protocol field. The experimental results show that the technique can parse out the protocol format correctly, especially for the encryption protocol, and extract the program behavior semantics of each protocol field.